6 research outputs found
Learning to Follow Instructions in Text-Based Games
Text-based games present a unique class of sequential decision making problem
in which agents interact with a partially observable, simulated environment via
actions and observations conveyed through natural language. Such observations
typically include instructions that, in a reinforcement learning (RL) setting,
can directly or indirectly guide a player towards completing reward-worthy
tasks. In this work, we study the ability of RL agents to follow such
instructions. We conduct experiments that show that the performance of
state-of-the-art text-based game agents is largely unaffected by the presence
or absence of such instructions, and that these agents are typically unable to
execute tasks to completion. To further study and address the task of
instruction following, we equip RL agents with an internal structured
representation of natural language instructions in the form of Linear Temporal
Logic (LTL), a formal language that is increasingly used for temporally
extended reward specification in RL. Our framework both supports and highlights
the benefit of understanding the temporal semantics of instructions and in
measuring progress towards achievement of such a temporally extended behaviour.
Experiments with 500+ games in TextWorld demonstrate the superior performance
of our approach.Comment: NeurIPS 202
Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machines
Natural and formal languages provide an effective mechanism for humans to
specify instructions and reward functions. We investigate how to generate
policies via RL when reward functions are specified in a symbolic language
captured by Reward Machines, an increasingly popular automaton-inspired
structure. We are interested in the case where the mapping of environment state
to a symbolic (here, Reward Machine) vocabulary -- commonly known as the
labelling function -- is uncertain from the perspective of the agent. We
formulate the problem of policy learning in Reward Machines with noisy symbolic
abstractions as a special class of POMDP optimization problem, and investigate
several methods to address the problem, building on existing and new
techniques, the latter focused on predicting Reward Machine state, rather than
on grounding of individual symbols. We analyze these methods and evaluate them
experimentally under varying degrees of uncertainty in the correct
interpretation of the symbolic vocabulary. We verify the strength of our
approach and the limitation of existing methods via an empirical investigation
on both illustrative, toy domains and partially observable, deep RL domains.Comment: NeurIPS Deep Reinforcement Learning Workshop 202
Resolving Misconceptions about the Plans of Agents via Theory of Mind
For a plan to achieve some goal -- to be valid -- a set of sufficient and necessary conditions must hold. In dynamic settings, agents (including humans) may come to hold false beliefs about these conditions and, by extension, about the validity of their plans or the plans of other agents. Since different agents often believe different things about the world and about the beliefs of other agents, discrepancies may occur between agents' beliefs about the validity of plans. In this work, we explore how agents can use their Theory of Mind to resolve such discrepancies by communicating and/or acting in the environment. We appeal to an epistemic logic framework to allow agents to reason over other agents' nested beliefs, and demonstrate how epistemic planning tools can be used to resolve discrepancies regarding plan validity in a number of domains. Our work shows promise for human decision support as demonstrated by a user study that showcases the ability of our approach to resolve misconceptions held by humans
Planning to Avoid Side Effects
In sequential decision making, objective specifications are often underspecified or incomplete, neglecting to take into account potential (negative) side effects. Executing plans without consideration of their side effects can lead to catastrophic outcomes -- a concern recently raised in relation to the safety of AI. In this paper we investigate how to avoid side effects in a symbolic planning setting. We study the notion of minimizing side effects in the context of a planning environment where multiple independent agents co-exist. We define (classes of) negative side effects in terms of their effect on the agency of those other agents. Finally, we show how plans which minimize side effects of different types can be computed via compilations to cost-optimizing symbolic planning, and investigate experimentally